DIRS.INFO The most relevant business & tech news

☆ Top websites block Google from training AI models on their data. Nowhere near as much as OpenAI, though.
▼ + stars: | 2024-03-14 | by ( Hugh Langley | ) www.businessinsider.com time to read: +4 min

Google launched a new tool that lets publishers opt out of training Google's AI models. It turns out that all this content has been stored in datasets that are the foundation for training powerful AI models, including those from OpenAI, Google, Meta, and others. Part of Google's response has been to launch a new tool that lets websites block the company from using their content for training AI models. BI asked Originality.ai CEO Jonathan Gillham why Google-Extended is being used less than other AI training data-blockers. It's unclear if the company will launch this fully in the future, or how much different it will be from the traditional Google search engine.

Persons: —, There's, Robots.txt, Jonathan Gillham, Gillham, Axel Springer Organizations: Google, Service, New York Times, CNN, BBC, Business Locations: Chicago

☆ The New York Times got its content removed from one of the biggest AI training datasets. Here's how it did it.
▼ + stars: | 2023-11-08 | by ( Alistair Barr | Kali Hays | ) www.businessinsider.com time to read: +4 min

The New York Times discovered a big AI training dataset contained links to its copyrighted content. The media company also found its content in other AI training datasets, such as WebText. The New York Times discovered that Common Crawl, one of the largest AI training datasets, contained millions of URLs linking to its paywalled articles and other copyrighted content. The New York Times has found its paywalled articles and other copyrighted content in other popular AI training datasets. It's unclear if The New York Times has managed to get its content removed from WebText and other AI training datasets.

Persons: —, OpenAI's, Google's Infiniset, Charlie Stadtlander, Masterclass, Kelly, GAI Organizations: New York Times, Service, The New York Times, Foundation, US, Amazon, Yorker, The Times Locations: Originality.ai

☆ OpenAI's GPTBot and other AI web crawlers are being blocked by even more companies now
▼ + stars: | 2023-09-28 | by ( Kali Hays | ) www.businessinsider.com time to read: +7 min

Unique, high quality data, mainly scraped from the web, is vital to the performance of AI models. AdvertisementAdvertisementMore and more companies are trying to avoid having their data freely scraped and saved by web crawlers working for the benefit of AI models. Last month, OpenAI last revealed its own crawler, GPTBot, saying it would respect robots.txt, a decades-old method through which a website can tell a web crawler to ignore it. Many more companies are now also blocking CCBot, a web crawler used by Common Crawl. AdvertisementAdvertisementSee below for a full list of the biggest websites now blocking GPTBot and CCBot as of Sept. 22:Blocking GPTBotamazon.comquora.comnytimes.comtheguardian.comshutterstock.comwikihow.comcnn.comsciencedirect.comusatoday.comhealthline.comstackexchange.comalamy.comscribd.comwebmd.combusinessinsider.comdictionary.comreuters.comwashingtonpost.commedicalnewstoday.comnpr.orgcbsnews.comgoodhousekeeping.comamazon.co.uktumblr.comlatimes.cominsider.comglassdoor.comvocabulary.cominvestopedia.comslideshare.netamazon.decosmopolitan.comnbcnews.comindiamart.comstackoverflow.comhindustantimes.combloomberg.comcnbc.compeople.comtvtropes.orgamazon.invimeo.comverywellhealth.comikea.comespn.comindianexpress.comthesaurus.compbs.org123rf.comwattpad.comvariety.comtoday.compopsugar.comthespruce.comuol.com.bramazon.frgeeksforgeeks.orgelle.comeconomictimes.compcmag.comtheverge.comallrecipes.comthoughtco.comrollingstone.comwired.comnextdoor.comhollywoodreporter.comabc.net.auew.comamazon.canews18.comwomenshealthmag.comrateyourmusic.comamazon.co.jptechradar.comairbnb.comndtv.comlifewire.comtomsguide.comvulture.comeverydayhealth.compolygon.comtheconversation.comesquire.comprnewswire.combillboard.commenshealth.commetro.co.ukcountryliving.commashable.comgamesradar.comthehindu.comtimesofindia.comdeadline.comharpersbazaar.commedscape.comnymag.comrefinery29.comradiotimes.comcbssports.comtandfonline.comtheatlantic.comtrulia.comamazon.espinterest.esnationalgeographic.combhg.comeater.comsouthernliving.comhealthgrades.comvice.compicclick.combustle.comnewyorker.comeonline.comdigitalspy.comopentable.compinterest.dethepioneerwoman.comcaranddriver.combyrdie.comlivemint.commedicinenet.comteacherspayteachers.comcookpad.comthespruceeats.combizjournals.compagesjaunes.frliputan6.comdelish.commasterclass.comarchiveofourown.orgvox.comrealsimple.comaarp.orgfrancetvinfo.frpinterest.frkumparan.comtheathletic.comtravelandleisure.comvogue.comlivescience.comapartments.commarketwatch.comglamour.comamazon.itcinemablend.comthrillist.comamazon.com.brpinterest.co.ukangi.comalamy.esusmagazine.comdistractify.combbcgoodfood.comjagran.commercadolibre.com.mxandroidauthority.comcity-data.comfoodandwine.comhellomagazine.comamazon.com.augq.comingles.comamarujala.comieee.orgprevention.comstern.dekbb.comedmunds.commarthastewart.compcgamer.comjustanswer.comhealth.com20minutes.frfortune.comhomes.comscientificamerican.compopularmechanics.comverywellfit.comvanityfair.comchicagotribune.comverywellmind.comhousebeautiful.comcntraveler.comallure.comspanishdict.comneverbounce.comanswers.commoneycontrol.comarchitecturaldigest.comslate.comlonelyplanet.cominverse.comcorriere.itactu.frself.comtripsavvy.cominstyle.comeatingwell.comsuperuser.comwelt.despiegel.dewomansday.comseventeen.comhbr.orgoprahdaily.comautotrader.combonappetit.comsueddeutsche.deseriouseats.comliveabout.comseattletimes.comcoursera.orglivehindustan.comfrance24.comtownandcountrymag.comdotesports.comworldplaces.mefaz.netteenvogue.commotor1.comnj.comglamourmagazine.co.ukokdiario.combrides.comstylecaster.comalamyimages.frjagranjosh.comtheglobeandmail.comaxios.comfrancebleu.frtabelog.comthebalancemoney.comnydailynews.comsheknows.comnaomedical.comverywellfamily.comBlocking CCBot

Persons: —, OpenAI, GPTbot, Conde Nast, Masterclass, Kelly, robots.txt, verywellhealth.com, indianexpress.com Organizations: Service, Amazon, Guardian, NPR, CBS News, CBS Sports, NBC News, CNBC, Yorker, Hearst, New York Times Locations: USA, Europe, Originality.ai, androidauthority.com

☆ AI is killing the grand bargain at the heart of the web. 'We're in a different world.'
▼ + stars: | 2023-08-30 | by ( Kali Hays | Alistair Barr | ) www.businessinsider.com time to read: +14 min

AdvertisementAdvertisementAI is undermining the web's grand bargain, and a decades-old handshake agreement is the only thing standing in the way. Now, though, generative AI and large language models are changing the mission of web crawlers radically and rapidly. Without a supply of potential consumers, there's little incentive for content creators to let web crawlers continue to suck up free data online. It's also open to manipulation, especially given the voracious appetite for quality AI data. Because robots.txt is voluntary, web crawlers can also simply ignore the blocking instructions and siphon the information from a site anyway.

Persons: Microsoft's Bing, Joost de Valk, It's, de Valk, Nick Vincent, Valk, OpenAI, robots.txt, Jason Schultz, Catherine Stihler, Archie, NYU's Schultz, Steven Sinofsky, who's, Andreessen Horowitz, De Valk, Stihler Organizations: Big Tech, Google, Wordpress, NYU's Technology, Policy Clinic, AWS, Creative Commons, Creative, Microsoft, Nvidia, Star Wars, DC Comics, Warner Brothers, Marvel, Disney, Atlantic, Meta Locations: CCBot, EleutherAI

☆ OpenAI just admitted it has a bot that crawls the web to collect AI training data. If you don't block GPTbot, that's self-sabotage.
▼ + stars: | 2023-08-08 | by ( Alistair Barr | ) www.businessinsider.com time to read: +8 min

Some of these bots have been helpful because they send users to sources of original content online. The most active one is probably Googlebot, which automatically collects web information so Google can later rank and serve it up in Search results. It's called GPTbot and it's being used to scrape and collect online content for AI model training. So what is Clarke's advice for other online content creators when it comes to GPTbot? What is the incentive that OpenAI offers to have these content creators allow GPTbot to crawl and scrape their sites?

Persons: OpenAI, Prasad Dhumal, Neil Clarke, Clarkesworld, Clarke, I've, hasn't Organizations: Morning, Twitter, OpenAI, Associated Press

Search resuls for: "CCBot"

5 mentions found